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PREFACE 


In  1994,  Congress  authorized  use  of  Other  Transactions  (OT)  for  the 
development  of  prototypes  “directly  relevant  to  weapons  or  weapon 
systems."^  Under  this  authority,  projects  are  not  required  to  comply 
with  procurement-specific  laws  and  regulations.  A  principal  objec¬ 
tive  of  the  legislation  was  to  enable  and  encourage  a  broader  range  of 
commercial  firms  to  participate  in  developing  defense  systems,  thus 
bringing  to  such  programs  similar  advanced  technologies  being  de¬ 
veloped  for  the  commercial  markets. 

During  the  1994-1998  time  period,  72  such  projects  began.  RAND 
was  asked  by  the  Department  of  Defense  (DoD)  to  assess  the  experi¬ 
ence  of  those  projects,  with  an  emphasis  on  implementation  and 
goal  achievement.  This  documented  briefing  presents  the  results  of 
that  assessment.  Our  results  should  be  of  interest  to  DoD  acquisition 
policymakers  and  analysts  in  government  and  industry. 

This  research  was  conducted  for  the  Director  of  Defense  Procure¬ 
ment  in  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  (USD[AT&L]),  within  the  Acquisition  and 
Technology  Policy  Center  of  RAND’s  National  Defense  Research 
Institute  (NDRI).  NDRI  is  a  federally  funded  research  and  develop¬ 
ment  center  sponsored  by  the  Office  of  the  Secretary  of  Defense,  the 
Joint  Staff,  the  unified  commands,  and  the  defense  agencies. 


^“National  Defense  Authorization  Act  for  Fiscal  Year  1994,”  Pub.  L.  103-160,  Sect.  845. 
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SUMMARY 


In  1994,  Congress  authorized  use  of  OT  for  the  development  of 
prototypes  “directly  relevant  to  weapons  or  weapon  systems.” 
Under  this  authority,  projects  are  not  required  to  comply  with 
procurement-specific  laws  and  regulations.  In  effect,  OT  authority 
provides  a  blanket  waiver  of  laws  such  as  the  Truth  in  Negotiations 
Act  and  the  Competition  in  Contracting  Act,  and  regulations  such  as 
the  Federal  Acquisition  Regulation  (FAR),  and  the  defense 
supplement  to  the  FAR.  A  principal  objective  of  the  legislation  was  to 
enable  and  encourage  a  broader  range  of  commercial  firms  to 
participate  in  developing  defense  systems,  thus  bringing  to  such 
programs  similar  advanced  technologies  being  developed  for  the 
commercial  markets. 

During  the  1994-1998  time  period,  72  such  projects  began.  RAND 
was  asked  by  DoD  to  assess  the  experience  provided  by  those  pro¬ 
jects.  The  objective  of  this  assessment  was  to  determine  the  overall 
effectiveness  of  this  new  acquisition  approach.  Specifically:  Do  the 
benefits  expected  from  relaxing  the  process  controls  justify  the  possible 
costs  that  might  be  incurred? 

We  conducted  this  assessment  by  performing  a  detailed  examination 
of  21  projects,  comprising  about  one-third  of  the  total  population 
that  had  accumulated  some  track  record  by  late  1999.  We  addressed 
four  topics: 

•  What  were  the  general  characteristics  of  typical  OT  agreements? 

•  What  benefits  appear  to  have  been  achieved  through  use  of  the 
OT  process? 
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•  Were  there  any  apparent  disadvantages  encountered,  and 
specifically  were  government  interests  appropriately  protected? 

•  What  were  the  net  effects  after  balancing  the  apparent  advan¬ 
tages  and  disadvantages? 

Our  assessment  of  OT  prototype  projects,  completed  in  March  2000, 
yielded  three  general  conclusions: 

Important  new  industrial  resources  are  now  participating  in  DoD 
prototype  projects  because  of  the  freedoms  inherent  in  the  OT  pro¬ 
cess.  However,  evidence  of  that  cannot  be  found  by  counting  “new" 
company  names  on  a  list  of  projects;  we  found  few  firms  with  no 
prior  DoD  contracting  experience  now  working  on  DoD  projects, 
with  most  of  those  at  the  subcontractor  level.  The  important  new  in¬ 
dustrial  capability  is  drawn  from  segments  of  major  firms  that  had 
been  focusing  exclusively  on  commercial  projects  but  are  now  will¬ 
ing  to  apply  their  skills  and  products  to  military  prototypes. 

The  benefits  of  OT  are  broader  than  just  the  addition  of  new  indus¬ 
trial  resources.  The  flexibility  of  the  OT  process  has  been  used  to: 
(a)  achieve  better  use  of  industry  resources  through  innovative 
business  arrangements  and  project  designs;  (b)  improve 
management  of  risks  and  uncertainties  through  freedom  to  modify 
the  program  as  it  evolves;  and  (c)  achieve  better  value  through  cost 
sharing  and  reduction  of  transaction  costs.  Overall,  more  effort  is 
being  devoted  to  product  and  less  to  process. 

Some  risks  to  the  government  are  incurred,  hut  we  believe  the  im¬ 
mediate  rewards  substantially  outweigh  the  risks.  Risks  arise 
mainly  through  relaxing  DoD  demands  for  access  to  the  firm  s  fi¬ 
nancial  records,  and  ownership  of  intellectual  property  (patents  and 
data).  However,  such  relaxation  of  DoD  rights  applies  to  only  a  few 
of  the  OT  prototype  projects,  mainly  those  involving  products  with 
strong  commercial  market  potential  and  where  the  firm  contributes 
a  significant  portion  of  the  development  resources.  Even  in  those 
few  projects,  we  believe  the  risks  to  DoD  are  limited.  Verification  of 
cost  records  becomes  relatively  unimportant  when  a  firm  is  con¬ 
tributing  a  large  share  of  project  costs.  Less-than-complete  govern¬ 
ment  ownership  of  intellectual  property  might  lead  to  increased 
costs  in  future  phases  of  the  project,  but  those  risks  are  limited.  This 
is  in  part  because  the  possible  future  costs  should  be  discounted  to 
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some  degree  as  they  are  in  the  future,  and  in  part  because  they  typi¬ 
cally  apply  in  areas  where  the  technology  is  moving  fast  and  where 
the  value  of  specific  kinds  of  knowledge  can  rapidly  decay.  Further¬ 
more,  if  the  flexibility  in  negotiating  intellectual  property  and 
financial  audit  clauses  is  removed  from  the  OT  authority,  most  if  not 
all  of  the  new  industrial  resources  would  again  become  unavailable  to 
DoD. 

These  conclusions  are  mostly  subjective  and  interpretive.  They  do, 
however,  represent  the  views  of  a  broad  cross  section  of  DoD  and  in¬ 
dustry  project  managers  and  agreement  officers  who  have  been 
managing  OT  prototype  projects.  The  rewards  of  creating  and  man¬ 
aging  a  more  efficient  and  effective  program  structure  and  manage¬ 
ment  process  have  led  those  participants  to  be  uniformly  enthusias¬ 
tic  about  the  OT  process  and  also  advocates  for  its  further  use. 
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Symbol 

COSSI 

DARPA 

DD21 

Dem/Val 

DFARS 

DoD 

DUAP 

EELV 

FAR 

GEO 

IR&D 

MIDS 

MMC 

NCAT 

NDRI 


Definition 

Commercial  Operations  and  Support  Savings 
Initiative 

Defense  Advanced  Research  Projects  Agency 

New  Navy  ship  program 

Demonstration /Validation 

Defense  Federal  Acquisition  Regulation 
Supplement 

Department  of  Defense 

Dual-Use  Applications  Program 

Evolved  Expendable  Launch  Vehicle 

Federal  Acquisition  Regulation 

Geostationary  orbit 

Independent  research  and  development 

Multifunctional  Information  Distribution 
System 

Metal  Matrix  Composite 

National  Center  for  Applied  Technology 

National  Defense  Research  Institute 
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NIMA 

National  Imagery  and  Mapping  Agency 

NML 

National  Media  Laboratory 

OT 

Other  Transactions 

OTA 

Other  Transactions  Authority 

UAV 

Unmanned  Air  Vehicle 

UCAV 

Unmanned  Combat  Air  Vehicle 

u.s.c. 

United  States  Code 

USD(AT&L) 

Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics 

VTOL 

Vertical  Takeoff  and  Landing 

INTRODUCTION 


OT  Is  a  Departure  from 
Standard  Procurement  Practice 

•  Law  allows  procurement  of  prototypes 
“directly  relevant  to  weapons  or  weapon 
systems”  without  complying  with 
procurement-specific  laws 

-  72  projects  started  from  1994  through  1998 

•  Expected  to  improve  government  access 
to  affordabie  commercial  technologies 


I  NDRI 


RAND 


Two  consistent  themes  in  the  Department  of  Defense's  (DoD's)  ac¬ 
quisition  reform  efforts  over  the  last  several  decades  have  been: 
streamlining  the  process  by  reducing  the  burden  caused  by  regula¬ 
tions  and  oversight  procedures  and  adopting  commercial  practices 
and  products.  One  reason  to  adopt  commercial  practices  was  to 
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broaden  the  direct  participation  of  commercial  industry  on  DoD 
projects.  The  thought  was  that  if  DoD’s  business  processes  matched 
commercial  practices,  an  important  set  of  factors  that  create  disin¬ 
centives  for  commercial  industry  participation  on  DoD  projects 
would  be  removed.  In  1994,  Congress  authorized  the  use  of  Other 
Transactions  Authority  (OTA)  for  the  development  of  prototypes 
directly  relevant  to  weapon  system  programs.^  The  authority  to  use 
Other  Transactions  (OT)  was  initially  given  only  to  the  Defense 
Advanced  Research  Projects  Agency  (DARPA);  several  years  later,  it 
was  extended  to  the  military  services  and  other  defense  agencies. 
OTA  is  a  powerful  reform  intended  to  further  DoD’s  objectives  of  im¬ 
proving  the  efficiency  of  the  acquisition  process  and  encouraging  the 
direct  participation  of  commercial  industry. 

DoD’s  access  to  commercial  industry  has  become  increasingly  im¬ 
portant.  DoD  no  longer  dominates  the  market  in  several  key  tech¬ 
nology  areas,  particularly  information  technology  and  systems.  The 
result  is  that  many  technological  advances  are  occurring  in  the 
commercial  market.  Under  traditional  approaches  to  acquisition, 
DoD  does  not  have  access  to  these  innovations;  commercial  firms 
consider  DoD's  business  processes  to  be  too  burdensome  to  warrant 
direct  participation  and  its  traditional  acquisition  rules  require  pass 
through  of  such  regulatory  constraints  to  subcontractors.  Of  particu¬ 
lar  concern  to  commercial  industry  were  cost  accounting  standards 
and  processes,  requirements  for  competition,  lack  of  use  of  perfor¬ 
mance-based  specifications,  and  intellectual  property  rights. 

At  the  same  time,  a  contracting  defense  industry  hurting  from  the 
decline  in  DoD  development  and  procurement  funds  was  lobbying 
strongly  for  the  relaxation  of  regulations  perceived  as  burdensome 
and  costly.  Defense  industry  was  advocating  a  more  common  sense 
approach  to  DoD  acquisition  that  would  enable  companies  to  be 
more  efficient  and  effective  in  their  use  of  resources. 

OT  provides  a  blanket  waiver  of  the  laws  and  regulations  that 
underpin  the  concerns  of  both  the  commercial  and  the  defense  in¬ 
dustry.  Processes  normally  required  by  the  Truth  in  Negotiations 
Act,  the  Competition  in  Contracting  Act,  the  Federal  Acquisition 


Uhe  legislative  history,  intent,  and  concepts  underlying  the  origins  and  use  of  Section 
845  OTA  are  described  in  detail  in  Kaminski  (1996)  and  Dunn  (1995;  1996a, b). 
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Regulation  (FAR),  and  the  Defense  Federal  Acquisition  Regulation 
Supplement  do  not  need  to  be  adhered  to.  Instead,  OT  allows  gov¬ 
ernment  and  industry  to  define  their  relationship  through  negotia¬ 
tions  without  the  normal  constraints.  Under  OT  authority,  the  pro¬ 
gram  management  approach,  program  objectives  and  criteria  for 
measuring  progress,  oversight  and  reporting  requirements,  price  to 
the  government,  fee  for  industry,  and  the  statement  of  work  are  em¬ 
bodied  in  an  Agreement  reflecting  the  results  of  this  negotiation. 
Agreements  are  inherently  more  flexible  than  traditional  contracting 
vehicles,  and  could  be  changed  simply  by  the  mutual  agreement  of 
government  and  industry  participants.  The  quality  of  an  Agreement 
is  more  closely  associated  with  the  skills  and  experience  of  the  gov¬ 
ernment  and  industry  managers  and  contracting  officers  than  a 
traditional  FAR-based  contract.  OT  usually  gives  the  contractor 
greater  program  management  responsibility  and  authority,  but  also 
gives  the  government  greater  insight  into  contractor  decisions  and 
status.  Rather  than  the  adversarial  relationship  between  government 
and  industry  under  traditional  processes,  OT  requires  a  more  collab¬ 
orative  working  relationship. 

Because  program  management  under  OT  can  potentially  remove  the 
traditional  oversight  and  accountability  processes,  critics  have  been 
concerned  that  DoD’s  interests  might  not  be  adequately  protected. 
In  particular,  the  lack  of  traditional  cost  accounting  and  auditing 
procedures  could  expose  DoD  to  greater  financial  risk.  Lack  of  own¬ 
ership  of  the  intellectual  property  resulting  from  OT  development 
activities  could  constrain  future  innovation  or  impose  future  costs  on 
the  government  because  of  limits  on  licensing  the  technology.  Simi¬ 
larly,  the  lack  of  technical  data  obtained  under  OT  could  adversely 
affect  DoD’s  ability  to  support  systems  using  that  technology.  These 
are  serious  concerns  that  should  be  addressed  before  OT  is  more 
widely  applied. 

In  lanuary  2001,  nine  months  after  the  completion  of  this  research, 
DoD  published  new  guidelines  for  the  use  of  OT  that  addressed 
many  of  the  concerns  expressed  by  critics.  The  guidelines  contain 
language  regarding  when  OT  use  is  appropriate,  the  definition  of 
nontraditional  firms,  and  auditing  procedures  that  could  have  ad¬ 
verse  affects.  That  language  is  taken  directly  from  the  statute  (10 
U.S.C.  2371,  Section  845),  which  was  revised  in  the  FYOl  National 
Defense  Authorization  Act.  There  are  indications  that  some  of  this 
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language,  particularly  regarding  the  conditions  under  which  OT  can 
be  used,  may  be  having  adverse  (and  unintended)  consequences  by 
reducing  the  opportunities  for  DoD  to  gain  access  to  commercial 
technologies  and  developments.  In  light  of  this,  we  believe  that  DoD 
should  conduct  further  evaluations  on  the  impact  of  the  guidelines 
on  OT  use  and  benefits. 

The  research  reported  here  describes  the  OT  implementation  experi¬ 
ence  to  date  and  assesses  its  costs  and  benefits.  In  the  course  of  this 
research,  we  also  begin  to  address  the  critics’  concerns  regarding 
financial  risk  and  intellectual  property. 

The  next  section  presents  the  main  briefing  documenting  the  results 
of  the  research.  A  subsequent  section  contains  a  set  of  backup  charts 
that  provide  more  detail  on  the  sample  of  projects  we  examined. 


RESEARCH  RESULTS 


RAND  Was  Asked  to  Assess  DoD 
Experience  with  OT  for  Prototypes 

•  Overarching  question:  Have  resuits  justified 
the  exceptionai  freedom  from  mandated 
compiiance  with  extensive  ruies  on: 

-  Management  oversight? 

-  Fiscal  accountability? 

-  Ownership  of  intellectual  property? 

-  Negotiation  of  contract  terms? 

-  Etc.? 


f  rand 


The  1994  congressional  authorization  to  use  OT  for  prototype  pro¬ 
jects  is  potentially  one  of  the  most  powerful  reforms  to  the  acquisi¬ 
tion  process  in  a  long  time.  In  one  action  it  swept  away  several 
decades'  accumulation  of  laws  and  procedures  that  constrained 
managers  in  how  they  designed  and  managed  such  projects  as  well 
as  how  the  government-industry  relationship  was  defined.  While  the 


6  Assessing  the  Use  of  “Other  Transactions”  Authority  for  Prototype  Projects 


intent  is  never  spelled  out  explicitly  in  congressional  documents,  it  is 
apparent  that  one  major  goal  was  to  improve  DoD  access  to  tech¬ 
nologies  that  were  being  developed  for  the  commercial  market.  We 
now  have  several  years  of  experience  with  the  use  of  OT  authority  for 
prototype  projects,  enough  to  make  an  initial  assessment  of  how  the 
process  is  working. 

In  fall  1999,  RAND  was  asked  to  perform  an  assessment  of  OT  experi¬ 
ence  for  prototype  programs.  While  there  are  many  detailed  ques¬ 
tions  and  issues  involved  in  assessing  OT  experience,  we  believe 
most  can  be  subsumed  under  one  overarching  question:  Have  the 
results  justified  the  authority  to  perform  projects  without  being 
required  to  comply  with  the  historical  accumulation  of  laws  and  pro¬ 
cedures?  Our  primary  research  objective  was  to  address  that  issue. 
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Our  Study  Examined  21  OT  Projects 
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Our  strategy  was  to  examine  a  sample  of  21  projects  in  some  detail, 
nearly  one-third  of  the  72  prototype  programs  started  during  the 
1994-1998  time  period.  Our  sample  includes  all  of  the  very  large  OT 
prototype  projects  and  a  good  distribution  of  the  medium  and 
smaller  projects. ^  We  did  not  examine  any  of  the  very  small  ones. 
Whether  this  exclusion  introduced  any  bias  in  our  conclusions  is  un¬ 
known. 

Our  sample  includes  a  wide  range  of  system  types  as  well  as  several 
programs  each  from  DARPA,  the  National  Imagery  and  Mapping 
Agency,  and  the  three  military  services.  In  each  project  we  examined 
the  agreements  and  associated  documents  carefully,  and  we  inter¬ 
viewed  project  managers  in  both  DoD  and  industry.  The  informa¬ 
tion  we  gathered  through  these  discussions  was  amazingly  consistent 
across  the  various  programs.  That  consistency  encourages  us  to 


^Project  size  was  measured  in  then-year  dollars. 
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believe  that  our  results  are  representative  of  the  overall  OT  experi¬ 
ence  with  prototype  programs,  but  it  is  not  absolutely  conclusive. 
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Study  Results  Were  Constrained  by 
Difficulties  in  Measuring  OT  Effects 

•  Too  many  variables,  too  few  programs  for 
quantitative  analysis 

•  Traditional  metrics — cost  growth,  schedule 
slip,  performance  shortfall— are  inappropriate 
for  prototype  projects 

•  Cannot  measure  path  not  taken 

•  Most  projects  not  yet  completed 

•  Research  results  are  qualitative  and  not 

rigorously  provable  _ 


One  important  element  of  our  research  was  to  develop  a  set  of  met¬ 
rics  that  would  measure  the  relative  effects  of  OT  on  program  out¬ 
comes  and  OT's  broader  policy  goals.  While  attempting  to  accom¬ 
plish  this,  we  were  unable  to  develop  any  practical  quantifiable 
metrics  that  others  would  find  credible.  The  few  quantifiable  metrics 
we  uncovered  are  either  misleading  (e.g.,  the  number  of  nontradi- 
tional  contractors)  or  unverifiable  (e.g.,  cost  avoidance).  This  result 
affects  both  the  kind  of  information  we  can  present  and  the  kind  of 
conclusions  that  can  be  drawn. 

The  reasons  we  did  not  find  a  practical  approach  to  such  metrics  are 
classic  in  the  field  of  acquisition  policy  analysis.  We  cannot  perform 
a  statistical  comparison  of  a  group  of  OT  programs  versus  a  group  of 
conventional  programs  because  there  are  too  many  variables  and  too 
few  programs.  Furthermore,  the  traditional  metrics  for  such  a  com¬ 
parison,  and  where  we  have  a  good  historical  database,  are  elements 
such  as  cost  growth  and  schedule  slip;  those  are  inappropriate  for 
prototype  programs  that  are  inherently  risky. 
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It  is  not  practical  to  compare  a  single  OT  program  with  a  counterpart 
conducted  under  traditional  contracting  methods  because  we  never 
have  an  analog  program  that  is  remotely  comparable.  We  do  not 
have  data  on  what  the  program  would  have  been  like  under  tradi¬ 
tional  methods.  And  finally,  most  of  the  programs  in  our  rather  small 
sample  are  still  under  way  so  we  do  not  have  true  outcomes. 

Thus,  we  rely  largely  on  qualitative  information  in  this  analysis — the 
judgments  and  opinions  of  experienced  managers  who  have  run 
both  kinds  of  programs.  We  are  unable  to  analytically  prove  the 
validity  of  those  judgments  and  opinions,  for  all  the  reasons  noted 
here.  For  those  same  reasons,  critics  are  unable  to  disprove  the 
claimed  benefits  of  OT  agreements  or  to  quantitatively  demonstrate 
the  superiority  of  another  policy  or  process. 


Research  Results  11 


\Ne  Explored  Four  Topics 

•  Characteristics  of  typicai  OT  agreements 

•  Benefits  of  OT  process 

•  Protecting  DoD  interests 

•  Conciusions 

[  ran D  | 


We  present  our  results  in  four  sections:  a  short  description  of  typical 
OT  agreements,  followed  by  a  discussion  of  the  major  benefits  that 
we  see  flowing  from  the  OT  authority;  we  then  address  the  major 
criticisms  of  the  process,  followed  by  our  general  conclusions. 

Throughout  this  research  we  were  generously  granted  access  to 
information  that  the  firms  considered  proprietary  or  commercially 
sensitive.  To  enable  unrestricted  distribution  of  this  document,  we 
avoid  reporting  any  such  information  here,  including  source  attribu¬ 
tion  to  certain  expressions  of  opinion  or  linking  some  specific  result 
to  a  specific  project. 
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OT  Agreements  Differ  from  Standard 
Contracts 

•  Relatively  short  documents  (10-30  pages) 

•  Each  clause  reflects  a  negotiated  position 
tailored  to  needs  of  all  parties 

-  FAR/DFARS  used  as  a  tool  kit,  not  a  rigid  format 


NDRI 


RAND 


OT  agreements  are  different  from  conventional  contracts  in  several 
important  ways.  One  is  that  they  are  relatively  short  and  clearly  writ¬ 
ten  without  use  of  specialized  legal  language.  But  probably  the  most 
important  fundamental  difference  is  that  each  and  every  clause  in  an 
agreement  represents  a  negotiated  position  tailored  to  the  needs  of 
all  parties.  The  essence  of  an  OT  agreement  is  that  there  are  no  rigid 
rules  that  must  be  applied,  which  opens  an  enormous  range  of  op¬ 
portunities  for  innovative  strategies  and  processes. 

Those  writing  the  agreements  are  thoroughly  versed  in  the  FAR  and 
associated  legislation,  and  they  use  these  procedures  as  a  framework 
when  designing  an  agreement.  In  most  ways  the  agreements  are 
consistent  with  traditional  contracts,  but  they  differ  in  a  few  impor¬ 
tant  ways.  We  identified  six  special  features  that  are  typical  of  OT 
agreements  and  that  lead  to  a  very  different  relationship  between 
DoD  and  industr>\ 
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OT  Agreements  Are  Collaborative 
Partnerships  Between  DoD  and  Industry 

•  Deliverable  is  typically  best  effort  toward  a  specified 
system  performance  goal 

•  Prototype  development  is  divided  into  several  incremental 
milestones 

•  All  terms  may  be  modified  by  mutual  agreement,  including 
canceling  the  project 

•  Each  party’s  financial  obligation  is  limited  to  a  specified 
amount 

•  Prime  contractor  has  complete  responsibility  and  authority 
for  managing  the  project 

•  Industry  frequently  shares  costs  with  DoD 


While  each  agreement  is  unique  and  reflects  the  needs  of  that  par¬ 
ticular  project,  as  well  as  the  preferences  of  the  managers  and 
agreement  officers,  we  find  that  certain  key  features  are  reflected  in 
most  agreements.  Taken  together,  these  features  define  a  new  rela¬ 
tionship  between  DoD  and  industry,  one  emphasizing  a  collabora¬ 
tive  partnership  where  all  parties  share  a  common  set  of  objectives: 

•  Reflecting  the  uncertainty  that  is  typical  of  most  development 
projects,  the  objectives  are  stated  in  terms  of  goals  rather  than 
fixed  requirements,  and  the  industry  team  is  obligated  only  to 
deliver  its  best  effort  toward  achieving  those  goals.  The  lure  of 
future  business  flowing  from  a  successful  prototype  is  a  powerful 
incentive  to  high  performance. 

•  Almost  every  project  is  structured  as  a  sequential  series  of  incre¬ 
mental  steps,  each  culminating  in  a  milestone  review.  Successful 
completion  of  a  milestone  typically  triggers  payment  of  a  speci¬ 
fied  amount  to  the  contractor  and  authorization  to  proceed  with 
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the  next  incremental  step.  In  some  cases,  each  milestone  is 
treated  as  a  fixed-price  task,  thus  eliminating  the  need  to  verify 
actual  costs  incurred. 

•  Prototype  development  projects  are  not  highly  predictable,  and 
some  modifications  to  the  original  plan  are  usually  needed  as 
experience  accumulates.  Such  changes  can  be  made  by  agree¬ 
ment  of  all  parties,  with  little  or  no  administrative  overhead 
other  than  to  document  the  change  and  make  corresponding 
adjustments  in  future  plans  and  schedules.  Typically,  no  exter¬ 
nal  review  or  approval  is  required  as  long  as  the  change  does  not 
affect  the  overall  project  goals  or  the  total  project  costs.  Such  a 
change  may  even  include  canceling  the  project;  this  occurred  in 
one  of  the  projects  we  examined. 

•  Each  party's  financial  obligations  are  stated  at  the  beginning, 
and  neither  party  has  any  obligation  to  spend  more  than  its 
specified  amount. 

•  The  contractor  management  team  has  full  authority  to  manage 
the  project  in  a  manner  and  style  it  considers  best.  The  DoD 
program  management  staff  participates  in  management  deci¬ 
sions  and  frequently  contributes  to  solving  the  problems  at  hand, 
but  the  staff  has  no  authority  to  direct  any  action  by  the  contrac¬ 
tor  unless  it  is  treated  as  a  change  in  scope,  with  a  corresponding 
change  in  projected  costs. 

•  Finally,  industry  frequently  makes  an  explicit  financial  contribu¬ 
tion  to  the  project.  This  occurs  even  when  the  product  has  a 
purely  military  application  but  the  industry  team  considers  it  an 
opportunity  to  gain  important  skills  that  might  contribute  to 
future  projects.  Industry  contributions  often  come  from  inde¬ 
pendent  research  and  development  (IR&D)  accounts. 

Taken  together,  these  agreement  elements  provide  a  powerful  degree 
of  flexibility  to  manage  the  development  of  projects  that  involve  a 
considerable  degree  of  uncertainty  and  risk.  They  also  define  a  very 
different  government- contractor  relationship  from  that  typically 
found  in  traditional  programs. 
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l/l/ie  Explored  Four  Topics 

•  Characteristics  of  typicai  OT  agreements 

•  Benefits  of  OT  process 

•  Protecting  DoD  interests 

•  Conclusions 


Next  we  will  outline  the  major  benefits  that  appear  to  have  been 
achieved  through  use  of  the  OT  authority. 
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Benefits  Are  Broader 
than  Expansion  of  industry  Base 

•  Originally  anticipated  benefit  is  being  achieved 
to  a  limited  extent 

-  Private  industry  more  wiiiing  to  work  for  DoD 

•  Other  important  benefits  are  emerging 

-  Better  management  of  risks  and  uncertainties 

-  Better  project  structure  through  new  and  innovative 
business  reiationships 

-  DoD  gets  more  value  per  dollar  spent 


We  noted  earlier  that  one  of  the  motivations  for  the  original  enabling 
legislation  was  to  expand  the  industry  base  available  to  DoD.  We 
find  that  goal  is  being  achieved  to  at  least  a  limited  degree.  We  also 
find  that  several  other  important  benefits  are  being  achieved  as 
managers  in  both  DoD  and  industry  learn  to  utilize  the  opportunities 
inherent  in  the  OT  process.  Each  of  these  four  benefits  is  addressed 
on  the  following  figures. 
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Benefits  Are  Broader 
Than  Expansion  of  industry  Base 

•  Originally  anticipated  benefit  is  being  achieved 
to  a  limited  extent 

-  Private  industry  more  wiiiing  to  work  for  DoD 

•  Other  important  benefits  are  emerging 

-  Better  management  of  risks  and  uncertainties 

-  Better  project  structure  through  new  and  innovative 
business  relationships 

-  DoD  gets  more  value  per  dollar  spent 


At  the  beginning  of  this  study  we  spent  a  lot  of  time  trying  to  find 
evidence  that  new  companies  had  entered  the  ranks  of  DoD  suppli¬ 
ers.  We  found  a  few.  But  we  finally  realized  that  a  major  part  of  the 
new  activity  comes  from  segments  of  large  firms  where  the  firm  is  a 
traditional  supplier:  names  like  3M,  Lucent,  Motorola,  Eastman 
Kodak,  Oracle,  and  others.  But  major  segments  of  those  firms,  using 
corporate  funds  to  develop  products  for  the  commercial  market,  had 
previously  been  unwilling  to  work  for  DoD  under  the  traditional 
contracting  process.  Now  those  broader  segments  of  the  firms  are 
willing  to  work  for  DoD  under  an  OT  agreement.  At  Motorola — to 
cite  one  specific  example  that  was  quoted  to  us — one  group  contain¬ 
ing  about  200  engineers  works  for  DoD  under  a  FAR-type  contract, 
but  under  an  OT  agreement  it  has  access  to  the  entire  corporate 
development  staff.  The  key  difference  is  that,  under  an  OT  agree¬ 
ment,  those  firms  can  protect  their  intellectual  property — a  topic  we 
discuss  later  in  this  briefing. 
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Our  search  for  new  firms  with  no  prior  experience  working  for  DoD 
only  led  to  one  prime  contractor,  Frontier  Systems,  and  that  example 
must  be  caveated  because  the  senior  managers  in  the  firm  are  thor¬ 
oughly  familiar  with  traditional  DoD  and  FAR-type  contracts,  but 
they  refuse  to  accept  the  associated  regulatory  burdens.  There  are  a 
few  new  firms  appearing  as  specialized  suppliers  (e.g.,  a  surfboard 
manufacturer  building  the  wings  of  the  Unmanned  Combat  Air 
Vehicle),  but  even  those  are  exceptions  to  the  rule. 

The  use  of  OT  agreements  is  achieving  the  goal  of  making  major 
segments  of  the  industry  available  to  DoD  for  the  first  time.  Demon¬ 
strating  that  cannot  be  done  by  simply  counting  new  names  on  a  list 
of  contracts  and  instead  requires  a  series  of  individual  company 
studies.  This  also  illustrates  the  difficulty  we  had  in  developing 
meaningful  quantitative  metrics. 
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OT  Allows  DoD  to  Better  Manage 
Risks  and  Uncertainties 

•  Change  and  adaptability  are  built  into  the 
process 

-  Plan  can  be  altered  by  mutual  agreement  with  minimal 
administrative  burden 

•  Deliverable  is  frequently  best  effort 

-  No  obiigation  to  deiiver  to  a  predetermined  specification 
if  that  proves  impracticai 

-  Incentives  used  instead  of  fixed  obiigation 
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Another  major  benefit  of  the  OT  process  is  the  improved  ability  to 
manage  the  risks  and  uncertainties  that  are  inherent  in  any  devel¬ 
opment  project.  OT  agreements  typically  contain  a  clause  stating 
that  any  time  progress  or  results  indicate  that  a  change  would  be 
beneficial  to  project  objectives,  such  a  change  can  be  made  through 
mutual  agreement  of  DoD  and  industry  managers.  As  long  as  the 
change  does  not  affect  the  stated  program  goals,  or  the  total  cost, 
those  changes  do  not  require  external  review  and  can  become  effec¬ 
tive  immediately  upon  agreement,  with  minimal  documentation 
internal  to  the  project.  Such  flexibility  provides  powerful  opportuni¬ 
ties  to  better  cope  with  the  problems  and  opportunities  that  occur 
when  developing  new  systems  and  components. 

The  authority  to  change  also  extends  to  the  premature  termination  of 
a  project  if  all  parties  agree.  In  one  of  the  projects  we  studied,  the 
project  was  terminated  at  the  halfway  point  because  progress  was 
unsatisfactory  and  effective  corrective  action  seemed  unlikely.  In 
another  case,  where  three  firms  were  competing  during  the  concept 
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design  phase,  the  DoD  program  manager  advised  one  firm  that  it  was 
far  behind  the  other  two  and  should  drop  out,  which  it  did  well 
before  formal  downselect.  We  found  many  other  examples  of  man¬ 
agement  flexibility  during  our  study. 

Another  powerful  feature  of  most  OT  agreements  is  that  the  objec¬ 
tives  are  stated  in  terms  of  general  goals  rather  than  detailed  product 
specifications,  thus  allowing  maximum  freedom  to  adjust  the  final 
design  to  match  emerging  knowledge  of  the  options.  Instead  of 
commitment  to  meet  specific  product  specifications,  a  variety  of  in¬ 
centives  are  incorporated  in  the  agreements  that  are  generally  built 
around  progress  payments  contingent  on  successful  completion  of  a 
milestone  event. 
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OT  Enables  Innovative  Business 
Relationships 

•  DoD  can  negotiate  with  competitive  firms  prior  to 
award 

•  DoD  can  more  easily  support  creation  of  industry 
partnerships  and  consortia  to  better  match  project 
needs 

•  DoD  can  pay  contractors  for  successfully  meeting 
established  milestones  rather  than  reimbursing  them 
for  incurred  costs 

-  Some  milestones  can  be  fixed  price 

•  DoD  can  work  directly  with  subcontractors  and 
consortia  members 
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OT  agreements  provide  an  opportunity  to  create  innovative  business 
arrangements  that  suit  the  particular  needs  of  the  individual  project. 
We  found  numerous  examples  during  our  survey. 

DoD  program  managers  can  negotiate  independently  with  individual 
firms  in  a  competitive  environment  to  improve  the  quality  of  the 
proposal  (e.g.,  a  more  focused  statement  of  work)  prior  to  award. 

In  several  instances,  the  initial  solicitation  yielded  proposals  showing 
that  a  partnership  of  two  or  more  firms  would  likely  yield  a  better 
product.  The  DoD  managers  could  negotiate  with  the  firms  and 
solicit  a  new  proposal  from  such  a  partnership  tvithout  the  lengthy 
process  of  preparing  and  advertising  a  new  solicitation.  Sometimes 
an  industry  partnership  is  more  easily  formed  through  informal  con¬ 
sortia,  with  one  firm  being  the  contract  agent.  An  OT  agreement 
offers  opportunities  for  such  arrangements  that  would  be  prohibited 
under  standard  contracting  rules. 
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While  a  few  of  the  OT  agreements  call  for  reimbursement  of  incurred 
costs,  most  tie  progress  payments  to  satisfactory  completion  of 
milestone  events.  In  a  few  cases  those  milestones  become  incremen¬ 
tal  fixed-price  obligations,  thus  obviating  the  need  for  verifying  costs 
incurred.  Such  an  arrangement  can  be  especially  attractive  when  the 
product  is  expected  to  have  a  strong  commercial  market. 

The  OT  agreement  offers  the  DoD  program  manager  an  opportunity 
to  become  directly  involved  in  project  activities  at  both  the  prime 
and  subcontractor  levels.  This  enables  the  program  manager  to  have 
a  much  better  understanding  of  the  true  project  status  than  in  a  typi¬ 
cal  program  where  the  prime  contractor  controls  the  flow  of  infor¬ 
mation  to  the  customer. 


Research  Results  23 


OT  Procedures  Can  Yield  More 
Value  per  DoD  Dollar  Invested 


•  Leverage  private  investment  through  sharing 
of  costs,  risks,  and  benefits 

•  Reduce  transaction  and  overhead  costs 

-  Vest  more  authority  in  government/industry 
management  team 

-  involve  small,  sequential  steps,  with  obligations  limited 
at  each  step 

-  Use  IR&D  rates,  other  commercial  practices 

•  Participants  claim  significant  cost  avoidance 
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The  OT  process  appears  to  provide  more  value  per  dollar  invested  by 
DoD.  This  was  the  single  most  common  metric  used  to  describe  the 
benefits  of  OT,  but  it  is  extremely  difficult  to  verify  the  claims  of  gov¬ 
ernment  and  industry  program  managers.  Nevertheless,  the  logic 
and  mechanisms  underlying  those  claims  appear  valid.  Thus,  while 
we  cannot  provide  a  quantitative  estimate,  we  believe  that  DoD  is 
obtaining  more  value  per  dollar.  This  is  usually  presented  in  terms  of 
relatively  increased  engineering  labor  hours  applied  per  dollar  spent. 

One  of  the  obvious  mechanisms  is  through  direct  sharing  of  the  costs 
and  risks,  as  well  as  the  benefits,  by  the  contractor.  Another  way  is 
through  reduced  transaction  and  overhead  costs.  Some  such  bene¬ 
fits  accrue  from  the  direct  involvement  of  DoD  managers  and  the 
consequent  lesser  amount  of  formal  oversight  and  reporting  required 
in  such  projects;  other  benefits  flow  from  the  incremental,  step-by- 
step  process  that  is  typically  employed  in  the  prototype  develop¬ 
ment,  with  a  consequent  reduction  in  risk-mitigation  measures  by 
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the  contractor;  and  in  some  cases  the  ability  to  use  IR&D  funds,  with 
their  lower  overhead  burden,  or  other  commercial  practices  leads  to 
lower  program  costs. 

The  full  consequences  of  such  cost  avoidance  measures  are  impos¬ 
sible  to  document;  as  discussed  earlier,  there  is  no  “reference  case” 
for  comparison.  We  rely  on  the  impressions  of  experienced  man¬ 
agers  in  both  DoD  and  industry  who  strongly  assert  that  such  effects 
are  real,  based  on  their  accumulated  experience  with  other  projects 
conducted  under  standard  procedures.  One  manager  of  an  espe¬ 
cially  complex  program,  with  multiple  competitive  contractors, 
asserted  that  he  had  completed  the  project  at  half  the  cost  and  half 
the  time  that  would  have  been  required  under  normal  contracting 
procedures.  The  accumulated  weight  of  such  expert  opinion  is  per¬ 
suasive. 
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We  Explored  Four  Topics 

•  Characteristics  of  typicai  OT  agreements 

•  Benefits  of  OT  process 

•  Protecting  DoD  interests 

•  Conclusions 
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We  now  turn  to  the  third  main  topic:  protection  of  DoD  interests  in 
an  OT  project. 
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Critics  Say  DoD  Interests  Are  Not 
Adequately  Protected 

•  Agreements  limit  DoD  access  to  financial 
records 

-  Reduce  DoD's  ability  to  verify  costs 

•  Agreements  limit  DoD  rights  to  inteiiectual 
property 

-  Reduce  future  ownership  of  inventions  and  data 
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It  is  not  surprising  to  find  some  criticism  of  any  process  that  appears 
to  circumvent  procedures  based  on  several  decades  of  experience. 
We  find  most  such  criticisms  focus  on  two  points:  (1)  that  some 
agreements  limit,  to  varying  degree,  direct  DoD  access  to  financial 
records  in  order  to  verify  that  contractor  costs  have  been  properly 
stated,  and  (2)  that  some  agreements  limit  the  DoD  rights  to  intellec¬ 
tual  property  flowing  from  the  project.  When  examining  these  criti¬ 
cisms  we  find  it  useful  to  categorize  the  projects  along  two  dimen¬ 
sions  (next  figure):  the  potential  commercial  market  for  the  product 
and  who  owns  the  intellectual  property  critical  to  its  development. 
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Criticisms  Appiy  to  Oniy 
a  Portion  of  the  OT  Programs 
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When  categorized  along  those  lines  we  find  that  the  criticisms  of  OT 
are  relevant  to  only  a  special  subset  of  OT  prototype  programs.  In 
the  classical  military  combat  system  development  there  is  little 
commercial  market  for  the  product,  and  the  firms  doing  the  devel¬ 
opment  bring  to  the  process  intellectual  property  already  owned  by 
the  government  because  it  was  developed  in  prior  military  develop¬ 
ments.  We  find  that  the  OT  prototype  programs  that  fall  in  the  lower 
left  corner  of  the  graph  typically  comply  closely  with  FAR- type  pro¬ 
visions  dealing  with  financial  records  and  intellectual  property. 
There  is  no  basis  for  criticizing  these  projects  because  of  relaxed 
oversight  controls. 

The  interesting  cases  are  those  that  fall  in  the  upper  right  corner  of 
the  graph.  Here  the  firms  expect  the  main  market  for  the  product  to 
be  in  the  commercial  sector,  and  they  bring  to  the  project  skills  and 
resources  that  mostly  have  been  developed  with  corporate  funds, 
and  frequently  involve  trade  secrets  of  considerable  value.  It  is  not 
surprising  that  the  firms  would  insist  on  retaining  control  over  that 
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information.  But  corporate  resources  of  this  type  are  typically  the 
things  the  OT  process  was  designed  to  make  available  to  DoD: 
cutting-edge  technologies  in  the  information  processing  area  where 
DoD  does  not  have  to  pay  the  full  development  costs.  Applying  clas¬ 
sic  regulatory  oversight  controls  to  projects  of  this  type  would  most 
probably  eliminate  such  projects,  thus  largely  nullifying  one  of  the 
main  advantages  of  the  OT  process's  design.  Furthermore,  only  a 
relatively  small  fraction  of  the  programs  we  studied  exercised  the  full 
range  of  exemptions,  thus  further  reducing  the  overall  exposure  of 
the  government  to  possible  damage  stemming  from  relaxed  over¬ 
sight  controls. 
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In  Those  Few  Programs,  Rewards  to 
DoD  Are  Expected  to  Outweigh  Risks 


Prior,  Industry-Funded  Development 


rand  I— 


In  this  special  category  of  projects  we  expect  that  the  long-term 
benefits  to  DoD  will  outweigh  the  risks,  but  we  cannot  prove  that 
assertion.  The  main  uncertainty  involves  the  future  consequences  of 
DoD  having  only  limited  rights  to  intellectual  property  (e.g.,  patents 
or  government  licenses  for  technologies,  data  rights  affecting  future 
operating  and  support  costs)  that  might  be  involved  in  future  devel¬ 
opment  projects  or  deployment  issues.  Lacking  full  rights  clearly 
represents  some  risk  of  future  costs.  However,  those  future  costs 
should  be  discounted  to  some  degree,  and  they  typically  apply  in 
areas  where  the  technology  is  moving  very  fast  and  where  the  value 
of  specific  kinds  of  knowledge  can  rapidly  decay. 

Against  those  risks  there  are  the  immediate  advantages  of  timely  and 
low-cost  access  to  the  kinds  of  technology  that  is  becoming  increas¬ 
ingly  important  to  DoD  (e.g.,  information  technology).  Right  now, 
the  immediate  rewards  appear  to  substantially  outweigh  the  risks  of 
future  costs. 
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We  Explored  Four  Topics 

•  Characteristics  of  typicai  OT  agreements 

•  Benefits  of  OT  process 

•  Protecting  DoD  interests 

•  Conclusions 


■■I  NDRI 


RAND 


Our  conclusions  are  briefly  summarized  on  the  following  two  figures. 
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Experience  to  Date  Is  Encouraging 

•  OT  process  is  providing  broad  benefits 

-  Access  to  commercial  developments  in  cutting-edge 
technologies 

-  innovative  business  relationships 

-  Flexibility  to  manage  risks 

-  More  value  per  dollar  invested 

•  Risks  to  government  appear  limited 

-  Compromise  made  in  only  a  few  programs,  in  trade  for 
other  benefits 

-  Long-term  value  of  those  trades  is  hard  to  estimate 


NDRI  RAND 


We  said  at  the  beginning  that  the  OT  process  is  potentially  one  of  the 
most  powerful  reforms  to  the  acquisition  process  that  has  occurred 
in  a  long  time.  We  find  the  results,  to  date,  on  the  21  projects  exam¬ 
ined  in  this  study  encouraging.  The  OT  process  is  providing  im¬ 
proved  access  to  important  commercial  technologies  as  well  as  im¬ 
proved  efficiencies  in  conducting  technologically  risky  prototype 
projects.  It  is  clear  that  as  DoD  managers  and  contracting  personnel 
became  familiar  with  OT  processes,  the  quality  of  the  agreements 
and  program  implementation  improved.  While  these  benefits  are 
impossible  to  quantify  in  an  analytically  rigorous  manner,  we  found 
a  strong  consensus  among  both  industry  and  government  managers 
that  these  procedures  can  be  beneficial  when  applied  to  the  appro¬ 
priate  projects  (described  in  the  next  figure). 

It  should  also  be  noted  that  to  achieve  the  full  benefits  inherent  in 
the  OT  process  requires  that  government  project  managers  and 
agreement  officers  exercise  some  initiative  and  skill  in  their  man- 
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agement  duties.  The  OT  process  allows  them  to  actively  manage  in¬ 
stead  of  following  routine  procedures. 

It  is  undeniable  that  the  relaxation  of  financial  and  other  controls  in¬ 
herent  in  the  OT  process  opens  some  opportunity  for  abuse.  How¬ 
ever,  the  process  also  strengthens  the  access  and  information  avail¬ 
able  to  government  managers,  thus  reducing  such  risks.  Further¬ 
more,  in  the  sample  we  examined,  reduced  oversight  controls  were 
limited  to  only  a  few  of  the  projects  and  even  there  they  appear  mod¬ 
est  in  comparison  with  the  benefits.  But  we  must  add  the  caveat  that 
those  risks  cannot  be  rigorously  assessed  at  this  time. 
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OT  Process  Appears  Appropriate  for 
Certain  Kinds  of  Projects 

•  Ones  offering  DoD  and  industry  an  opportunity  to  jointly 
deveiop  a  product  with  strong  future  market  potentiai 

-  Gives  DoD  access  to  commercially  driven  technology  with 
dual-use  applications 

•  Ones  with  considerabie  uncertainty  in  outcomes 

-  Provides  DoD  the  flexibility  of  best-effort  deliverable  and  the 
freedom  to  change  or  cancel  the  project 

•  Ones  offering  DoD  the  opportunity  to  benefit  from 
innovative  business  relationships 

-  Allows  DoD  and  industry  to  assemble  strong  corsortia  teams 


An  underlying  research  issue  was  an  attempt  to  determine  the  condi¬ 
tions  under  which  OT  application  makes  sense.  Our  research  to  date 
suggests  that  when  at  least  one  of  the  following  conditions  is  met,  OT 
will  likely  be  beneficial: 

•  When  DoD  desires  access  to  technology  that  is  predominantly 
the  result  of  commercial  development,  OT  provides  a  mecha¬ 
nism  for  nonintrusive,  value-added  participation. 

•  When  there  is  considerable  uncertainty  regarding  both  perfor¬ 
mance  goals  and  what  is  technically  achievable  and  affordable, 
OT  provides  the  necessary  flexibility  to  manage  high-risk  pro¬ 
jects. 

•  When  DoD  might  benefit  from  innovative  business  relationships 
with  industry,  or  among  industry  participants,  OT  provides  the 
mechanism  to  define  those  relationships. 


APPENDIX 


The  following  figures  provide  supporting  information  relevant  to  the 
study  but  not  included  in  the  main  briefing. 
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OT  Projects  Have  Represented  Less  Than 
10  Percent  of  DoD  Dem/Va!  Spending 


1994  1995  1996  1997  1998 


DoD  Dem/Va!  Funding  1994-1998 


We  have  seen  very  limited  usage  of  OT  to  date.  The  projects  that 
were  started  under  an  OT  agreement  during  the  1994-1998  time 
period  have  a  cumulative  value  of  slightly  more  than  $1  billion, 
which  is  less  than  10  percent  of  the  Demonstration/ Validation 
(Dem/Val)  budget  (budget  category  6.3B)  during  those  years. 

The  actual  portion  of  the  Dem/Val  budget  used  for  OT  projects  is 
somewhat  less  than  10  percent  because  some  of  those  projects  were 
conducted  under  the  Commercial  Operations  and  Support  Savings 
Initiative.  Those  projects  are  intended  to  leverage  private  sector 
research  and  development  by  inserting  leading-edge  commercial 
technologies  into  fielded  military  systems  to  reduce  operations  and 
support  costs.  Cost  sharing  is  mandated  (at  least  25  percent  must 
come  from  nonfederal  funds)  and  government  funds  are  frequently 
drawn  from  budget  categories  other  than  Dem/Val,  so  the  OT 
prototype  projects  represent  an  even  smaller  fraction  of  actual 
Dem/Val  projects  during  that  period. 
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Four  Case  Studies  Involved  Projects 
$1  Million  to  $10  Million  In  Size 


<$1M  $1M-10M  $10M-100M  >$100M 


Project  Funding  Size 


RAND  I 


This  and  the  following  two  figures  identify  by  name  the  projects 
included  in  the  sample  examined  during  this  study.  Note  that  the 
Evolved  Expendable  Launch  Vehicle  project  was  examined  in  less 
detail  than  the  others  because  of  constraints  in  data  availability. 


07  Projects  1994-98  Ji  OT  Projects  1994-98 

(Number)  §  (Number) 
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Eleven  Case  Studies  Involved  Projects 
$10  Million  to  $100  Million  in  Size 


30" 


25 


20 


;■  Projects  RAND  studied 


Geospatial  Registration 
of  Information 

NIMA  (NML) 


-|5  AffordaUe  Rapid 
Response  Missile 
Demonstrator 


10 

5 

0 

Rl  |i 


Silicon  Carbide  MMC 
Track  Shoe 

MIDS  Low-Vol  Terminal 
Production 


<$1M  $1M-10M  $10M-100M 

Project  Funding  Size 


VTOL  UAV 

Helicopter  Usage 
Monitor  (COSSI) 

Tactical  Common 
Data  Link 


Dragonfly  VTOL 

NIMA  (NCAT) 

GEO  Light 
Imaging  Testbed 


>$100M 


^  RAND  I 


Six  Case  Studies  invoived  Projects 
Exceeding  $100  Miiiion  in  Size 


Projects  RAND  studied  *  =  incompiete 

-  data 
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